Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I followed the official Databricks documentation("https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html")to integrate MongoDB Atlas with Spark by setting up the MongoDB Spark Connector and configuring the connection string in my Datab...
Hi @vidya_kothavale ,Could you try to change "spark.mongodb.input.uri" to following?spark.read.format("mongodb").option("spark.mongodb.read.connection.uri"
Hi everyone,I'm trying to query data from our adls2 delta lake using a serverless sql warehouse. We've already set up private connectivity via NCC, but hitting a snag when running queries like:SELECT * FROM delta.`abfss://container@xxx.dfs.core.windo...
Hi, thanks for the detailed explanation.Unfortunately, configuring fs.azure.account.key in the Serverless advanced options didn’t help (I’m sure I wrote it correctly) - still receiving the same error.I saw in some sources around the net that I should...
I am using streaming tables to read from multiple parquet files and create a table in my raw layer.Tech Stack -- dbt-databricksWhile loading I have a column called "ets" in my source which is nothing "milliseconds since Jan 1st 1970." I want this to ...
Hi @singh_tushar_14 ,Since your ets columns already contains integer that represents milliseconds since Jan 1st 1970, can't you just use Spark SQL fuction? You don't need to add anything to 1970-01-01, since that information is already "encoded" in y...
serverless compute:cannot find column index for attribute error We are seeing this error if there are a complex view, the same view runs fine in sql warehousebut break in serverlesss compute onlylooks like serverless compute has issue parse large co...
ok, I will test on latest version 4 and will let you know if any more issues, thanks
Hi All,I've encountered what seems like a very specific bug. For some reason, running a query with the following where clause results in an execution error.where lower(organization_name) like '%trac%'There's no execution error when trying "tra" and i...
@JPan I noticed you had the new SQL editor mode on in your video. For what it's worth, I flicked between both when testing and both were successful. Given that you're finding it's only happening on some fields, do you think this is to do with special...
What are the new features we are going to see in the Dashboard this year?
@szymon_dybczak thanks for those suggestions. I didn't even know about the Product Roadmap Webinar every quarter. I'll definitely be tuning in for those.All the best,BS
Hi All,I've been using notebooks to run patch requests to an external API using the Python requests library. Often times certain notebooks will randomly start to fail throughout the day and will raise a ImportHookFinder.find_spec() error when attempt...
Hey, Did you manage to solve this issue? I am experiencing the same error but under a different context of using pytest in a notebook.My scenario:import pytest import os import sys os.chdir(f'/Workspace/Shared/DataTechnology-Databricks-Core') # Skip...
Photon ran out of memory while executing this query. Photon failed to reserve 32.0 KiB for BufferPool, in BroadcastBufferedRelation(spark_plan_id=13407). Memory usage: Total task memory (including non-Photon): 1829.7 MiB BroadcastBufferedRelation(spa...
Hello @vijamit Good day!!It was very hard to analyise the error but : The causes for the error shared are : Your query has run out of memory during execution, specifically when using the BuildHashedRelation and PartitionedRelation functions.Running o...
Using the old SQL editor, there was an option to add tags to saved queries under the "edit query info" option shown in the screenshots. Since starting to use the new editor, I noticed that option has disappeared from its previous location and I haven...
Hello @jtbing!At the moment, adding or editing tags for saved queries isn’t available in the new SQL editor UI. There’s also an upcoming Bricktalks session with the Product Manager for Databricks SQL, introducing the new SQL editor; you may find it u...
Hi All,I am getting below message when hovering on the Create SQL warehouse. I can only see the Serverless Starter Warehouse compute.You have reached the maximum number of SQL warehouses. Delete an existing warehouse to create new one.Regards,RN
I want to create notebooks for creating and accessing lakehouse . But I want it to be created using the assistant. Do we have a step by step guide to do it?
Open the assistant and ask it to spin up a lakehouse notebook and it should drop in the code you can run right away to connect and explore. If you're thinking to download the notebook, make you download it in pdf format and rename it after download ...
How can you control whether a file is a Notebook or a Query? Long story short - can you do anything to make Databricks recognise a .dbquery.ipynb file as a query and open it in SQL Warehouse?I created a file in my git folder as a query. All good. ...
Hello @dylan19 Good day!!Thats very frustrating! well just to check Did you change the .dbquery.ipynb file to a .sql file in your Git repository. For example, extract the SQL content from the .ipynb file (a JSON format) and saved it as plain text in ...
We were working with Python scripts on serverless (Git folder). The following code worked when we were calling on custom modules in sitting in the python scripts (eg. preprocess data from preprocessing.py). We've moved to shared compute but can no lo...
It seems you are using a workspace path. To confirm are you using a dedicated compute with access provided to a group. You need to provide can mange access to workspace path to the group.
Hi community, As the title says, I have been checking the documentation, but I could not find clear references to my question.I am working in a project, and I need to change all the plots fonts to match a specific brand. All plots are generated with ...
Set plt.rcParams['font.family'] = 'YourBrandFontName' at the start and all your plots will follow that font automatically, most probably.
Hi!We're trying to use the choropleth visualisation in the new dashboard. This works well for all countries we tried except for district level of Finland.The supported districts are clearly stated in the CSV on the documentation page titled county-di...
Hi Matt, You are correct that today, the Choropleth maps are using sub-regions as defined in https://en.wikipedia.org/wiki/Sub-regions_of_Finland. We are currently getting these boundaries from Mapbox, which provides this boundary data. Definitel...
