博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
【Python fits】Error: Too many open files
阅读量:6098 次
发布时间:2019-06-20

本文共 1765 字,大约阅读时间需要 5 分钟。

循环打开了好多个Fits文件读数据,然后突然提示“你打开太多文件啦”,后来就加了一行hdu.close(), 但是并没有什么乱用

查了一下说明才发现,是因为用open()打开后是存储在mmap中的,为了缓解内存,关闭hdu也没有用,文件还是开着呢

而怎么关闭文件呢,需要等到numpy觉得你并不需要这个data之后“自动”关闭,啊,如果它一直觉得你要用。。。。嗯。。。

解决方法很简单,告诉它,“我不用这个data啦”

所以,不是加一句hdu.close(),而是del hdu_data     del hdu.data

测试了一下,这样解决的是fits文件比较大,里面包含了太多的hdu,但是对于fits文件比较多的情况不适用

fits文件太多的时候,用hdu = fits.open(fits_name,memmap=False)直接不存在mmap中

 

原文和链接在下面

Say you have some code like:

The details may differ, but the qualitative point is that the data to many HDUs and/or FITS files are being accessed in a loop. This may result in an exception like:

 

As explained in the , because Astropy uses mmap by default to read the data in a FITS file, even if you correctly close a file with  a handle is kept open to that file so that the memory-mapped data array can still be continued to be read transparently.

The way Numpy supports mmap is such that the file mapping is not closed until the overlying  object has no references to it and is freed memory. However, when looping over a large number of files (or even just HDUs) rapidly, this may not happen immediately. Or in some cases if the HDU object persists, the data array attached to it may persist too. The easiest workaround is to manually delete the .data attribute on the HDU object so that the  reference is freed and the mmap can be closed:

In some extreme cases files are opened and closed fast enough that Python’s garbage collector does not free them (and hence free the file handles) often enough. To mitigate this your code can manually force a garbage collection by calling  at the end of the loop.

In a future release it will be easier to automatically perform this sort of cleanup when closing FITS files, where needed.

 

转载于:https://www.cnblogs.com/clementinadoo/p/9675653.html

你可能感兴趣的文章
linux下telnet到h3c交换机Backspace无效
查看>>
修复/lib/ld-linux.so.2: bad ELF interpreter: No such file or directory问题
查看>>
如何将sqlserver的windows验证模式改为SQL Server 和 Windows 混合身份验证模式
查看>>
Windows Server 2012 R2 DirectAccess功能测试(1)—DC配置
查看>>
性能测试计算公式 供参考
查看>>
瞬间读懂什么是互联网思维、大数据、O2O、众筹、红海
查看>>
表闪回
查看>>
jQuery插件 -- Cookie插件jquery.cookie.js(转)
查看>>
常见EMC疑问及对策
查看>>
About Me
查看>>
BZOJ4756:[USACO]Promotion Counting(线段树合并)
查看>>
内部类、代码块
查看>>
软件工程第二章 习题2 第5题
查看>>
残缺的字符串【FFT】
查看>>
ZZULIOJ 1917: E
查看>>
南阳oj 题目2—括号配对问题
查看>>
C/C++的const区别
查看>>
定位Java运行时 代码段方法
查看>>
Eclipse 编译错误 Access restriction:The type *** is not accessible due to restriction on... 解决方案...
查看>>
python第三方库汇总
查看>>