- 1075 Views
- 10 replies
- 27 kudos
Resolved! Study Groups / Study Space
Hi everyone, Not sure how practical this idea is to implement but I'd love to have a space / area where there's voice channels. It could be something like an "Official Databricks Community" Discord? Alternatively, it could be something that the websi...
- 1075 Views
- 10 replies
- 27 kudos
- 27 kudos
I love love love the idea of community based study groups. We definitely have regional based in person meetups right now, these are organized by customers in the online regional groups that you can monitor on the events page. I also appreciate that y...
- 27 kudos
- 233 Views
- 1 replies
- 0 kudos
How to choose the right node type for my Databricks workload?
Hey all, I’m setting up a Databricks workspace to run data pipelines + ML training. I’m unsure which node types to pick (driver vs worker, instance size, memory vs CPU optimized) for different workloads. For example small ETL jobs vs large batch proc...
- 233 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Ednexllc ,It really depends on many factor like workload type, size of your data, number of tables etc. You can check some recommendations given by Databricks here:Compute configuration recommendations | Databricks on AWSAnd also here - you shoul...
- 0 kudos
- 301 Views
- 3 replies
- 2 kudos
Incremental refresh issue in materialized views
i am facing the refresh issue . i am a trying to create a mv from a simple select query (select* from table where client="A") and the source table is a delta live table (streaming). The refresh is full recompute not incremental .
- 301 Views
- 3 replies
- 2 kudos
- 2 kudos
@hidden as per @szymon_dybczak 's screenshot, it needs to be serverlessAll the best,BS
- 2 kudos
- 572 Views
- 6 replies
- 4 kudos
Resolved! Pass parameter value to sql query in databricks sql alert
Hi,I have created a sql query with the parameter.Now using that parameterized sql query in databricks legacy sql alert.And scheduling that databricks sql alert in databricks workflow/job using Sql task type.Now how should I pass the parameter value t...
- 572 Views
- 6 replies
- 4 kudos
- 7317 Views
- 15 replies
- 12 kudos
Chrome/Edge high memory usage for Databricks tabs.
Is it normal for Databricks tabs to be using such high memory?The Chrome example I just got a screenshot of was this (rounded up/down)...3 x Databricks tabs for one user, sized at6gb, 4.5gb, and 2gbTotal = 12.5gbI know it gets higher than this too, I...
- 7317 Views
- 15 replies
- 12 kudos
- 12 kudos
And the git folders have gotten progressively slower and slower as well, and sometimes the 'open in new tab' button requires several clicks to work. The whole web interface feels vibecoded now.
- 12 kudos
- 341 Views
- 2 replies
- 0 kudos
Structuring multi-hop architectures for sensor data normalization in energy platforms - Databricks
Energy platforms increasingly rely on high-frequency sensor telemetry to monitor assets, optimize performance, and drive predictive analytics. However, telemetry from field devices, substations, and distributed energy resources often arrives in incon...
- 341 Views
- 2 replies
- 0 kudos
- 315 Views
- 1 replies
- 0 kudos
Streaming anomaly detection in oil pipelines using ML models on Databricks Structured Streaming
Oil pipelines operate under extreme pressures and conditions, where even minor anomalies such as abrupt pressure shifts or irregular flow rates can signal leaks, blockages, or equipment failure. Traditional monitoring systems rely on batch data and s...
- 315 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Danial_Gohar,Thanks for sharing. One tip for you, next time if you have something you'd like to share with community we have dedicated place for that: Community Articles.
- 0 kudos
- 3492 Views
- 2 replies
- 0 kudos
Resolved! Unable to create a Managed table in Unity Catalog defaul location
We have setup the metastore with Manged Identity and when trying to create a managed table in the default location I am hitting below error. The storage is ADLS Gen2. AbfsRestOperationException: Operation failed: "This request is not authorized to pe...
- 3492 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi,I am facing the same issue. I am getting 403 error while creating a TABLE.The folder is getting created inside the metastore in the ADLS everytime I run the Create Table command.The permissions are `Storage Blob Data Contributor` role assigned a...
- 0 kudos
- 814 Views
- 6 replies
- 0 kudos
Error when executing spark.readStream Script
Hi all,When I try to execute the script (as per the screenshot below), in a Notebook cell, I get an error message. I am using Databricks Free Edition, and am not sure if the error relates to the compute cluster that I am using?Any guidance would be ...
- 814 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @giuseppe_esq , Can you please share the DBR version, cluster configurations and log4 file containing the error stact trace to further review?
- 0 kudos
- 1015 Views
- 7 replies
- 5 kudos
Resolved! Error when trying to Create a STREAMING TABLE using Databricks SQL
Hi all,I am currently using the Databricks Free Edition, and trying to create a streaming table, using the following SQL script:%sqlCREATE OR REFRESH STREAMING TABLE sql_csv_autoloaderSCHEDULE EVERY 1 WEEKASSELECT *FROM STREAM read_files( '/Volumes/...
- 1015 Views
- 7 replies
- 5 kudos
- 5 kudos
Yes - I was able to do it no problem on Free, with the exact steps above, using the default supplied compute.
- 5 kudos
- 783 Views
- 4 replies
- 1 kudos
Resolved! Writing to data to a .csv file ( in the Databricks free edition)
I have been testing out a notebook. Is is possible to write any tabular outputs to a .csv file and if so would this be the filesystem or an S3 bucket.I get errrors when I try either approaches with the snippet below.#sort by match date to get the mo...
- 783 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @ilir_nuredini ,Thanks very much for the advice as this really helped esp DFS not available on the free edtions.I ran into an errror and I realised I needed to create a volume and schema first. So with with a little help from the AI assistant, -...
- 1 kudos
- 1967 Views
- 5 replies
- 0 kudos
Access S3 Tables from Databricks
Hi, When I was looking for how to access S3 Tables from Databricks, I came across this AWS blog post: https://aws.amazon.com/blogs/big-data/access-amazon-s3-iceberg-tables-from-databricks-using-aws-glue-iceberg-rest-catalog-in-amazon-sagemaker-lakeho...
- 1967 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @zhouhaijia, Are you trying to access S3? What are you looking to solve in particular? All the best,BS
- 0 kudos
- 716 Views
- 2 replies
- 0 kudos
Issue with the SQL Server ingestion in Databricks Lakflow- Gateway Stuck at "Starting"
Context: I am working on a POC to explore Databricks Lakeflow ingestion feature. I have tried creating Ingestion pipeline with salesforce connector successfully and able to load the data into Delta table. However when I am trying to create ingestion ...
- 716 Views
- 2 replies
- 0 kudos
- 0 kudos
We were able to resolve the above issue by increasing the quotas for two separate compute families. One was the Standard F4s family, which apparently the worker uses and the other was the Standard EDV4 family which I guess is the driver node. Now we'...
- 0 kudos
- 319 Views
- 1 replies
- 2 kudos
Resolved! Need help in creating compute cluster
Hi everyone,I hope you’re doing well.I’m currently facing an issue while trying to create a cluster in Azure Databricks (Personal). Despite trying various approaches, I keep encountering the following error message:The VM size you are specifying is n...
- 319 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Naveen_Sequeira ,If you increased quota and it didn't help then just try to use different region. In Azure, some machines are sometimes unavailable due to regional capacity issue. They should be available in another region though.You can try with...
- 2 kudos
- 224 Views
- 1 replies
- 0 kudos
Passing a list of images on AI_Query
HiI would like to know how can I pass a list of images to ai_query on the same call, It will be helpful to extract some metadata information on a multipage document, where I want to know if different images to see if it belongs to the same document.T...
- 224 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Andreyai Has your issue got resolved?
- 0 kudos
