Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
Hello,I created a sql warehouse (cluster size = 2X-Small) and I wanted to use it to execute a query using the sql query api:- url : https://databricks-host/api/2.0/preview/sql/statements- params = {'warehouse_id': 'warehouse_id','statement': 'SELECT ...
@Yahya24 can you please remove preview in query, they are not in preview any more "/api/2.0/sql/statements/", you should see json response, can you please check drop down menu and change to json, some times it may be setted into text, but usual respo...
I've been doing some testing with Partitions vs Z-Ordering to optimize the merge process.As the documentation says, tables smaller than 1TB should not be partitioned and can benefit from the Z-Ordering process to optimize the reading process.Analyzin...
Hi @gmiguel, I've been testing with Partitions vs. Z-Ordering to optimize the merge process. As the documentation says, tables smaller than 1TB should not be partitioned and can benefit from the Z-Ordering process to optimize the reading process.Ana...
How do you handle reporting monthly trends within a data lakehouse? Can this be done with timetravel to get the table state at the end of each month or is it better practice to build a data warehouse with SCD types? We are new to databricks and lak...
@Mswedorske IMO it would be better to use SCD.When you do VACUUM on a table, it removes the data files that are necessary for Time Travel, so it's not a best choice to rely on Time Travel.
In an All-Purpose Cluster, it is pretty easy to get at the Driver logs. Where do I find the Driver Logs for a SQL Pro Warehouse? The reason I ask is because sometimes in a SQL Editor we get generic error messages like "Task failed while writing row...
Hi @BamBam , In Azure Synapse Analytics (formerly SQL DW), the driver logs are not directly accessible as they are in Databricks. However, you can access the error in SQL Pro, the driver node is not exposed to users since SQL Pro uses a managed serv...
I cancel my subscription plan (account created via accounts.databricks.com), without noticing that by doing this, all of my workspaces will be deleted. Is this possible to recover those workspace, and if possible, what should I do to recover them?
Hi @San2 , I'm sorry, but once a Databricks subscription is cancelled, all workspaces associated with that account are deleted and this deletion is not reversible. When you cancel your subscription, you can no longer access workspaces, notebooks, or ...
Hi everyone!Is there any function in Databricks's dashboards to pause/halt visualization before a filter is inputted on the dashboard? Currently, my queries/visualizations are getting too long because of the data, and I want to stop it before a filte...
Hi @RawSid, There is no explicit function to pause or halt visualization before a filter is applied in Databricks's dashboard. It's recommended to contact Databricks support for more accurate information.
Within our team, there are certain (custom) python packages we always use and import in the same way. When starting a new notebook or analysis, we have to import these packages every time. Is it possible to automatically make these imports available ...
@Kaz You can install these libraries using the Libraries section in the Compute. All of the libraries mentioned here would be installed whenever the cluster is spun up.
Is it possible to show the full logs of a databricks job? Currently, the logs are skipped with:*** WARNING: max output size exceeded, skipping output. ***However, I don't believe our log files are more than 20 MB. I know you can press the logs button...
Hi @Kaz , it's not possible to display the complete logs of a Databricks job in the job overview if the log output size has been exceeded. Databricks have a limit on the size of the output logs that can be displayed in the job overview. If the output...
Is there a way to create a calculated field in a dashboard from the data that has been put into it?I have an aggregated dataset that goes into a dashboard, but using an average in the calculation will only work if I display the average by the grouped...
Hi @SimonMcCor , Yes, you can create a calculated field in a dashboard from the data that has been put into it. In Databricks, you can perform this operation within the notebook that feeds data into your dashboard. If you want to request a new featur...
We have a setup where we process sensor data in databricks using pyspark structured streaming from kafka streams, and continuisly write these to delta tables. These delta tables are served through a SQL warehouse endpoint to the users. We also store ...
Hi @Erik , - Use Delta Lake to optimize tables by coalescing small files into larger ones- Use the OPTIMIZE command on Delta Lake tables to improve write speed and reduce the number of small files- Utilize ZORDER for multi-dimensional clustering to ...
Does anyone know when serverless SQL will be available for ap-southeast-1 region?
Hi @lawrence009 , Let me get back to you on this.
Getting this error in dbt when trying to run a query. Not happening in the actual SQL warehouse in Databricks. Is this a bug? Can only find source code when I search 'DatabricksSQLCursorWrapper' but no documentation or information otherwise.
Hi @colinsorensen, Based on the given information, it appears that the error you are encountering is related to accessing cloud storage in Databricks. The error message suggests that the cluster does not have the necessary permissions to access the s...
Does anybody know whether we could customise the dates showing in the calendar selection in sql editor/dashboard?My query has a time frame in a particular period, however when I use DateRange parameter in sql editor, it could allow users to choose th...
Hi @hehuan-yu-zen , Yes, you can customize the dates showing in the calendar selection in the SQL editor/dashboard. The Date Range, Date and Time Range, and Date and Time Range (with seconds) parameters now support the option to designate the startin...
Hi allWe are having some alarming issues regarding a script that yields different output when running on SQL vs Notebook. The correct output should be 8625 rows which it is in the notebook, but the output in Databricks SQL is 156 rows. The script use...
UPDATE:I think we have identefied and solved the issue. It seems like using LAST with Databricks SQL requires to excplicitly be careful about setting the "ignoreNull" argument and also be careful about the correct datatype. I guess this is because of...
We have previously been able to access our Databricks cluster in R using ODBC but it stopped working a couple of months ago and now i can't get it to connect.I've downloaded the latest drivers and added the right information in odbc/odbcinst files bu...
Hi @uberweiss, Based on the error message you are receiving, the issue might be related to the server host or port specified for the connection. Here are a few suggestions to troubleshoot this issue. 1. Verify Server Details: The error message sugges...
