data storage

Whether you're a newbie or an experienced programmer, any questions, help, or just talk of any language will be welcomed here.

Moderator: Coders of Rage

mattheweston
Chaos Rift Junior
Chaos Rift Junior
Posts: 200
Joined: Mon Feb 22, 2010 12:32 am
Current Project: Breakout clone, Unnamed 2D RPG
Favorite Gaming Platforms: PC, XBOX360
Programming Language of Choice: C#
Location: San Antonio,Texas
Contact:

data storage

Post by mattheweston »

I'm trying to figure out what to use for storing the data in my game. It's going to be a football game and I need to store all the stats as well as rosters and league information. I am currently debating between the following three options.

1. Roll my own
2. Access DB
3. SQLite

The game is going to have to create files that are able to be copied and ran on multiple computers like saving game data to a memory card and then using that on someone else's console.

Anyone ever tried/had to use a db for their game? Thoughts?
Image
User avatar
superLED
Chaos Rift Junior
Chaos Rift Junior
Posts: 303
Joined: Sun Nov 21, 2010 10:56 am
Current Project: Engine
Favorite Gaming Platforms: N64
Programming Language of Choice: C++, PHP
Location: Norway

Re: data storage

Post by superLED »

What are you programming in?
mattheweston
Chaos Rift Junior
Chaos Rift Junior
Posts: 200
Joined: Mon Feb 22, 2010 12:32 am
Current Project: Breakout clone, Unnamed 2D RPG
Favorite Gaming Platforms: PC, XBOX360
Programming Language of Choice: C#
Location: San Antonio,Texas
Contact:

Re: data storage

Post by mattheweston »

C#...I'm leaning more so toward using SQLite.
Image
User avatar
Falco Girgis
Elysian Shadows Team
Elysian Shadows Team
Posts: 10294
Joined: Thu May 20, 2004 2:04 pm
Current Project: Elysian Shadows
Favorite Gaming Platforms: Dreamcast, SNES, NES
Programming Language of Choice: C/++
Location: Studio Vorbis, AL
Contact:

Re: data storage

Post by Falco Girgis »

...Roll your own.
User avatar
LeonBlade
Chaos Rift Demigod
Chaos Rift Demigod
Posts: 1314
Joined: Thu Jan 22, 2009 12:22 am
Current Project: Trying to make my first engine in C++ using OGL
Favorite Gaming Platforms: PS3
Programming Language of Choice: C++
Location: Blossvale, NY

Re: data storage

Post by LeonBlade »

GyroVorbis wrote:...Roll your own.
This. There's no need for a database, you're better off making something yourself which does exactly what you need without all the unneeded SQL stuff which is slow and useless for a game.
There's no place like ~/
User avatar
lotios611
Chaos Rift Regular
Chaos Rift Regular
Posts: 160
Joined: Sun Jun 14, 2009 12:05 pm
Current Project: Game engine for the PC, PSP, and maybe more.
Favorite Gaming Platforms: Gameboy Micro
Programming Language of Choice: C++

Re: data storage

Post by lotios611 »

LeonBlade wrote:
GyroVorbis wrote:...Roll your own.
This. There's no need for a database, you're better off making something yourself which does exactly what you need without all the unneeded SQL stuff which is slow and useless for a game.
When should you use a database then?
"Why geeks like computers: unzip, strip, touch, finger, grep, mount, fsck, more, yes, fsck, fsck, fsck, umount, sleep." - Unknown
Aleios
Chaos Rift Cool Newbie
Chaos Rift Cool Newbie
Posts: 78
Joined: Mon Feb 21, 2011 2:55 am
Current Project: Aleios Engine
Favorite Gaming Platforms: PC, Dreamcast
Programming Language of Choice: C++
Location: Melbourne, Australia

Re: data storage

Post by Aleios »

lotios611 wrote:
LeonBlade wrote:
GyroVorbis wrote:...Roll your own.
This. There's no need for a database, you're better off making something yourself which does exactly what you need without all the unneeded SQL stuff which is slow and useless for a game.
When should you use a database then?
When in need of storing massive amounts of data. MMO's typically use databases, and all the crap for that is handled server side.

In this case he could simply make an XML file, or even just a text file. Chuck the data in some format in either a text file or XML file and it can be copied to many machines with ease.
Image
User avatar
dandymcgee
ES Beta Backer
ES Beta Backer
Posts: 4709
Joined: Tue Apr 29, 2008 3:24 pm
Current Project: https://github.com/dbechrd/RicoTech
Favorite Gaming Platforms: NES, Sega Genesis, PS2, PC
Programming Language of Choice: C
Location: San Francisco
Contact:

Re: data storage

Post by dandymcgee »

Aleios wrote:
lotios611 wrote:
LeonBlade wrote:
GyroVorbis wrote:...Roll your own.
This. There's no need for a database, you're better off making something yourself which does exactly what you need without all the unneeded SQL stuff which is slow and useless for a game.
When should you use a database then?
When in need of storing massive amounts of data. MMO's typically use databases, and all the crap for that is handled server side.

In this case he could simply make an XML file, or even just a text file. Chuck the data in some format in either a text file or XML file and it can be copied to many machines with ease.
That's just one of a million different places a database is used. Databases have a ton of advantages over a simple XML file. I would recommend using a database for any project require a large, indexed data store. If you need to search through thousands of records for a particular one, XML will perform like shit compared to SQLite. If you just need to store settings for one user, a text database (XML, BIN, TXT, whatever format it may be) will be more suited to your project.

@mathewweston In your case it depends one what you're doing with the league / roster information. I'd assume you're just loading it up for a particular "level" in the game then using it from there. In that case XML would work just fine.
Falco Girgis wrote:It is imperative that I can broadcast my narcissistic commit strings to the Twitter! Tweet Tweet, bitches! :twisted:
User avatar
Falco Girgis
Elysian Shadows Team
Elysian Shadows Team
Posts: 10294
Joined: Thu May 20, 2004 2:04 pm
Current Project: Elysian Shadows
Favorite Gaming Platforms: Dreamcast, SNES, NES
Programming Language of Choice: C/++
Location: Studio Vorbis, AL
Contact:

Re: data storage

Post by Falco Girgis »

Unless you are making an MMO or other sort of game where data is stored on a server, there is absolutely no need to use something as large and slow as a SQL database in an indie game.

And I also think XML is over-used. It's expensive and slow as hell to parse. It's overkill in almost every scenario that I've seen it used... If you are using a Level Editor or Toolkit, why in god's name would you go with ANYTHING other than a straight binary file? It's WAY smaller and WAY faster. Without debate, the optimal solution. The entire point of a Level Editor is to prevent you from having to create assets by hand... which is the only reason you would ever use something like XML.
User avatar
szdarkhack
Chaos Rift Cool Newbie
Chaos Rift Cool Newbie
Posts: 61
Joined: Fri May 08, 2009 2:31 am

Re: data storage

Post by szdarkhack »

GyroVorbis wrote:And I also think XML is over-used. It's expensive and slow as hell to parse. It's overkill in almost every scenario that I've seen it used... If you are using a Level Editor or Toolkit, why in god's name would you go with ANYTHING other than a straight binary file? It's WAY smaller and WAY faster. Without debate, the optimal solution. The entire point of a Level Editor is to prevent you from having to create assets by hand... which is the only reason you would ever use something like XML.
Because it compresses well :mrgreen:

Sorry, i just had to say it :lol: On a serious note though, it depends on the situation. If you need to store a lot of data and only read them just a few times, xml works fine (for example, a configuration file). But for stuff that you need to load quickly (like tilemaps) you're definitely better off with a binary file. It will save you a lot of loading time.
User avatar
LeonBlade
Chaos Rift Demigod
Chaos Rift Demigod
Posts: 1314
Joined: Thu Jan 22, 2009 12:22 am
Current Project: Trying to make my first engine in C++ using OGL
Favorite Gaming Platforms: PS3
Programming Language of Choice: C++
Location: Blossvale, NY

Re: data storage

Post by LeonBlade »

GyroVorbis wrote:Unless you are making an MMO or other sort of game where data is stored on a server, there is absolutely no need to use something as large and slow as a SQL database in an indie game.

And I also think XML is over-used. It's expensive and slow as hell to parse. It's overkill in almost every scenario that I've seen it used... If you are using a Level Editor or Toolkit, why in god's name would you go with ANYTHING other than a straight binary file? It's WAY smaller and WAY faster. Without debate, the optimal solution. The entire point of a Level Editor is to prevent you from having to create assets by hand... which is the only reason you would ever use something like XML.
I agree with Falco, really the only time you should be using a database is if the information is going to be pulled across the Internet aka an MMO. In your case it makes the most sense to just pack together a binary of what you need and then parse it out. You save space, you save time by not using SQL and having to construct queries to pull out small bits of data, and it fits your needs and nothing more or less.

I also agree about XML being overused. XML is really only useful when it comes to things that you'll be changing manually. XML is a markup language, it's useful for changing manually because it's easy to read. In your case, the only thing that should be modifying your files is your application/game and nothing else really.

I hate seeing people talk about how they made an XML file for their level editor or for whatever they needed. And then they have to go and parse through the XML elements to grab information either manually with matching the XML string and not actually using proper XML parsing, or getting an XML parser that can spit out an array that you can pull data from. All of that is really slow and pointless, honestly a huge waste of time.

If you want to prototype something, or play around, at least make a comma delimited text file or something, not full blown XML. Then later once you realize what you need in your files, you can create a file that you can read in and you wont have to worry about being shit deep in XML for all your files and now it comes time to make everything into a binary.
There's no place like ~/
User avatar
Falco Girgis
Elysian Shadows Team
Elysian Shadows Team
Posts: 10294
Joined: Thu May 20, 2004 2:04 pm
Current Project: Elysian Shadows
Favorite Gaming Platforms: Dreamcast, SNES, NES
Programming Language of Choice: C/++
Location: Studio Vorbis, AL
Contact:

Re: data storage

Post by Falco Girgis »

szdarkhack wrote:
GyroVorbis wrote:And I also think XML is over-used. It's expensive and slow as hell to parse. It's overkill in almost every scenario that I've seen it used... If you are using a Level Editor or Toolkit, why in god's name would you go with ANYTHING other than a straight binary file? It's WAY smaller and WAY faster. Without debate, the optimal solution. The entire point of a Level Editor is to prevent you from having to create assets by hand... which is the only reason you would ever use something like XML.
Because it compresses well :mrgreen:

Sorry, i just had to say it :lol: On a serious note though, it depends on the situation. If you need to store a lot of data and only read them just a few times, xml works fine (for example, a configuration file). But for stuff that you need to load quickly (like tilemaps) you're definitely better off with a binary file. It will save you a lot of loading time.
XML compresses better than a custom, binary-based file packed to 100% efficiency with not a single wasted character? And what about having to decompress this? And who says you can't compress your already compressed binary file?

And sure, XML works "fine" when you're storing a lot of data and only reading them a few times like a config file... but goddamn, including an entire XML parsing library to load one file seems like overkill to me. Call me old-fashioned and low level, but that's definitely not something I will agree with. XML definitely has its place in software development... but unless you don't have a Level Editor, I don't see it in game development...
LeonBlade wrote:I hate seeing people talk about how they made an XML file for their level editor or for whatever they needed. And then they have to go and parse through the XML elements to grab information either manually with matching the XML string and not actually using proper XML parsing, or getting an XML parser that can spit out an array that you can pull data from. All of that is really slow and pointless, honestly a huge waste of time.
THANK YOU. I think this is the pathological worst-case. What is the point of a Level Editor? To automatically generate assets for your game/engine. Why the hell would you ever mess with these files manually? You are gaining absolutely nothing good from using XML in this scenario. It requires an additional runtime dependency in both your engine and editor, slows down your loadtimes, and bloats the hell out of your files. You've added NOTHING useful.

With a Level Editor, you have a prime opportunity to compress the hell out of your files and make everything as efficient and tightly-packed as possible.
User avatar
szdarkhack
Chaos Rift Cool Newbie
Chaos Rift Cool Newbie
Posts: 61
Joined: Fri May 08, 2009 2:31 am

Re: data storage

Post by szdarkhack »

I was kidding about the xml compressability... It's just what "fans" say all the time and i'm sick of hearing it. Using it for configuration purposes is in my opinion perfectly reasonable if you want them to be human-readable, given that the amount/structure of data justifies it. For example, if you can do it with a name-value pair file then do it, it'll be faster. There are cases, however, when you might need to have a more complex, hierarchical structure in your stored data, in which case XML is a pretty fair choice. In general though, the simpler you can afford to make things, the faster/smaller they will be.

Again, i'm in no way implying or suggesting that people should use xml for game data, you're clearly better off with binary files there. I thought i made that clear, but here it is once more, just to make sure.
User avatar
LeonBlade
Chaos Rift Demigod
Chaos Rift Demigod
Posts: 1314
Joined: Thu Jan 22, 2009 12:22 am
Current Project: Trying to make my first engine in C++ using OGL
Favorite Gaming Platforms: PS3
Programming Language of Choice: C++
Location: Blossvale, NY

Re: data storage

Post by LeonBlade »

szdarkhack wrote:I was kidding about the xml compressability... It's just what "fans" say all the time and i'm sick of hearing it. Using it for configuration purposes is in my opinion perfectly reasonable if you want them to be human-readable, given that the amount/structure of data justifies it. For example, if you can do it with a name-value pair file then do it, it'll be faster. There are cases, however, when you might need to have a more complex, hierarchical structure in your stored data, in which case XML is a pretty fair choice. In general though, the simpler you can afford to make things, the faster/smaller they will be.

Again, i'm in no way implying or suggesting that people should use xml for game data, you're clearly better off with binary files there. I thought i made that clear, but here it is once more, just to make sure.
Yeah I knew you were joking. ;) :lol:
There's no place like ~/
User avatar
Falco Girgis
Elysian Shadows Team
Elysian Shadows Team
Posts: 10294
Joined: Thu May 20, 2004 2:04 pm
Current Project: Elysian Shadows
Favorite Gaming Platforms: Dreamcast, SNES, NES
Programming Language of Choice: C/++
Location: Studio Vorbis, AL
Contact:

Re: data storage

Post by Falco Girgis »

LeonBlade wrote:
szdarkhack wrote:I was kidding about the xml compressability... It's just what "fans" say all the time and i'm sick of hearing it. Using it for configuration purposes is in my opinion perfectly reasonable if you want them to be human-readable, given that the amount/structure of data justifies it. For example, if you can do it with a name-value pair file then do it, it'll be faster. There are cases, however, when you might need to have a more complex, hierarchical structure in your stored data, in which case XML is a pretty fair choice. In general though, the simpler you can afford to make things, the faster/smaller they will be.

Again, i'm in no way implying or suggesting that people should use xml for game data, you're clearly better off with binary files there. I thought i made that clear, but here it is once more, just to make sure.
Yeah I knew you were joking. ;) :lol:
I didn't. :oops:

Myyy bad.
Post Reply