Strange 'open file dialog' problem (C++, VS)

[superLED]

Aw. I hate making new topics every time I stumble upon a new problem that I simply can't figure out myself.
We really should have a 'Quick questions' topic in here, to avoid all of the 'I have a small problem' topics.

Anyway, I have come across a problem that really kills my Map Editor. And the root of the problem is the Windows API's 'open file dialog' function.

I use the file dialog to get the path of the tilesheet I want to add. After that, I use the returned string to load that tilesheet into the program.
But when I do certain actions after using this file dialog, I get a crash. It doesn't matter what I use the returned string for; even when opening the file dialog but do nothing about the returned string, it will still crash (only when I select a file, and not clicking 'cancel').

This is the function I have created.
Have in mind that I am NOT familiar with the Windows API, so this is mostly copy-and-paste.

Code: Select all

// Declaration (window.h)
static string getFileName(char *filter = "PNG files (*.png)\0*.png\0\0", HWND owner = NULL);

// Defenition (window.cpp)
string Window::getFileName(char *filter, HWND owner) {
	OPENFILENAME ofn;
	char fileName[MAX_PATH] = "";

	ZeroMemory(&ofn, sizeof(ofn));

	ofn.lStructSize = sizeof(OPENFILENAME);
	ofn.hwndOwner = owner;
	ofn.lpstrFilter = filter;
	ofn.lpstrFile = fileName;
	ofn.nMaxFile = MAX_PATH;
	ofn.Flags = OFN_EXPLORER | OFN_FILEMUSTEXIST | OFN_HIDEREADONLY;
	ofn.lpstrDefExt = "";

	string fileNameStr;

	if (GetOpenFileName(&ofn)) {
		fileNameStr = fileName;
	}

	return fileNameStr;
}

// In use
string str;

str = Window::getFileName();

Here is the result of the crash:


A little background information
I have a small scripting system in my editor. I click 'Save script', and the new script is stored into the system. This works perfectly fine.
I also have a Tilesheet system, where users can click 'Add tilesheet', and the tilesheet is stored and can be used to draw the map.
The way I get the desired tilesheet into the system, is to open a file dialog, so I can browse to the tilesheet. This file dialog returns the name of the file as a string. Then, I use this string to point to the path where I should load up the new tilesheet from.
But after I have added a tilesheet (with the help of the file dialog), or simply just open a file dialog, selecting a file and not doing anything with the returned string, and then adding a new script, the program crashes.
The same happens when I try to pop up some text to the screen, after having the file dialog open.

[short]

can you post the typedef for OPENFILENAME plz

[superLED]

can you post the typedef for OPENFILENAME plz

CommDlg.h

Code: Select all

typedef OPENFILENAMEA OPENFILENAME;

... And in another location in the wall of text:

CommDlg.h

Code: Select all

typedef struct tagOFNA {
   DWORD        lStructSize;
   HWND         hwndOwner;
   HINSTANCE    hInstance;
   LPCSTR       lpstrFilter;
   LPSTR        lpstrCustomFilter;
   DWORD        nMaxCustFilter;
   DWORD        nFilterIndex;
   LPSTR        lpstrFile;
   DWORD        nMaxFile;
   LPSTR        lpstrFileTitle;
   DWORD        nMaxFileTitle;
   LPCSTR       lpstrInitialDir;
   LPCSTR       lpstrTitle;
   DWORD        Flags;
   WORD         nFileOffset;
   WORD         nFileExtension;
   LPCSTR       lpstrDefExt;
   LPARAM       lCustData;
   LPOFNHOOKPROC lpfnHook;
   LPCSTR       lpTemplateName;
#ifdef _MAC
   LPEDITMENU   lpEditInfo;
   LPCSTR       lpstrPrompt;
#endif
#if (_WIN32_WINNT >= 0x0500)
   void *        pvReserved;
   DWORD        dwReserved;
   DWORD        FlagsEx;
#endif // (_WIN32_WINNT >= 0x0500)
} OPENFILENAMEA, *LPOPENFILENAMEA;

[Nokurn]

You showed your locals but not your call stack, which may contain useful information in tracking down the bug.

Code: Select all

static Window::string getFileName(char *filter = "PNG files (*.png*)\0*.png*\0", HWND owner = NULL);

Your default filter looks dodgy. According to OPENFILENAME on MSDN:

lpstrFilter
Type: LPCTSTR
A buffer containing pairs of null-terminated filter strings. The last string in the buffer must be terminated by two NULL characters.

So your filter should be:

Code: Select all

"PNG files (*.png)\0*.png\0\0"

Also, what is the type Window::string? I have tested your code with std::string (as follows) with no error. Try this code in a fresh project and see if it crashes. If the problem cannot be reduced and reproduced in a clean project, it most likely lies elsewhere. Unsafe use of global or static instances can lead to crashes in the most unrelated of places. Same goes for buffer overflows.

Code: Select all

#include <Windows.h>
#include <string>

using namespace std;

string getFileName(const char *filter, HWND owner)
{
    char fileName[MAX_PATH] = "";

    OPENFILENAME ofn;
    ZeroMemory(&ofn, sizeof ofn);
    ofn.lStructSize = sizeof ofn;
    ofn.hwndOwner = owner;
    ofn.lpstrFilter = filter;
    ofn.lpstrFile = fileName;
    ofn.nMaxFile = MAX_PATH;
    ofn.Flags = OFN_EXPLORER | OFN_FILEMUSTEXIST | OFN_HIDEREADONLY;
    ofn.lpstrDefExt = "";

    string fileNameStr;
    if (GetOpenFileName(&ofn))
        fileNameStr = fileName;
    return fileNameStr;
}

int WINAPI WinMain(HINSTANCE hInst, HINSTANCE hPrevInst, LPSTR lpCmdLine, int nCmdShow)
{
    //string str = getFileName("PNG files (*.png*)\0*.png*\0", NULL);
    string str = getFileName("PNG files (*.png)\0*.png\0\0", NULL);
    MessageBox(NULL, str.c_str(), "getFileName", MB_OK);
    return 0;
}

[superLED]

Also, what is the type Window::string?

I kinda messed up the declaration and definition in my post. It's actually:

window.h: static string getFileName(char *filter = "PNG files (*.png)\0*.png\0\0", HWND owner = NULL);
window.cpp: string Window::getFileName(char *filter, HWND owner) {

And I fixed the filter as you said.

This is the call stack at crash:

[avansc]

kinda just glanced over, but make sure you show the backtrace of ALL threads. Im sure the win API spawns threads out the wazoo, so the bt you pasted might not be the culprit.


If I had to take a wild stab in the dark, it would be that you are giving it an uninitialized/unmalloced/too small/whatever var/mem, and its trying to do things with it. This might be off base as I mentioned I did not read much of the post. Just step through the frames and look at what is going on. BadPtr sounds like a dangling pointer, IE: not initialized.

OR, your "open file dialog" thingy is returning a NULL pointer for some reason and you are not checking for it then using it, and then pachow you blow up.... Technically speaking of course.