Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
I am having issues with Datatbricks SQL and dbt at the moment. When running a query a string column is somehow converted to a number. Does anybody have any idea why this would be happenning?
Hi, currently we are using Power BI as the semantic layer because it allows us to build custom measures to do aggregates and business logic calculation, and provides native connection to Excel. I am thinking to move these logics to Databricks using S...
Thanks @Retired_mod , I think Power Query can only perform pre-data transformations, not on-the-fly calculations in response to user filters.
Using Delta Sharing connector with Power BI, does it only works for import and currently no support for direct query?
@scrimpton currently it only supports import https://learn.microsoft.com/en-us/power-query/connectors/delta-sharing
Hi,I'm implementing a Databricks connector using the ODBC driver and currently working on the functionality to Cancel an ongoing SQL statement.However, I can't seem to find any ODBC function or SQL function to do so.The only other alternative I see i...
I am trying to connect to DBeaver from Databricks and getting this error message:[Databricks][DatabricksJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: javax.net.ssl.SSLHandshakeException: PKIX path building faile...
I have the same issue after upgrading cluster to DBR 12.2. Working fine with DBR 10.4
Hello,I created a sql warehouse (cluster size = 2X-Small) and I wanted to use it to execute a query using the sql query api:- url : https://databricks-host/api/2.0/preview/sql/statements- params = {'warehouse_id': 'warehouse_id','statement': 'SELECT ...
@Yahya24 can you please remove preview in query, they are not in preview any more "/api/2.0/sql/statements/", you should see json response, can you please check drop down menu and change to json, some times it may be setted into text, but usual respo...
I've been doing some testing with Partitions vs Z-Ordering to optimize the merge process.As the documentation says, tables smaller than 1TB should not be partitioned and can benefit from the Z-Ordering process to optimize the reading process.Analyzin...
I've been doing some testing with Partitions vs Z-Ordering to optimize the merge process.As the documentation says, tables smaller than 1TB should not be partitioned and can benefit from the Z-Ordering process to optimize the reading process.Analyzin...
How do you handle reporting monthly trends within a data lakehouse? Can this be done with timetravel to get the table state at the end of each month or is it better practice to build a data warehouse with SCD types? We are new to databricks and lak...
@Mswedorske IMO it would be better to use SCD.When you do VACUUM on a table, it removes the data files that are necessary for Time Travel, so it's not a best choice to rely on Time Travel.
In an All-Purpose Cluster, it is pretty easy to get at the Driver logs. Where do I find the Driver Logs for a SQL Pro Warehouse? The reason I ask is because sometimes in a SQL Editor we get generic error messages like "Task failed while writing row...
In an All-Purpose Cluster, it is pretty easy to get at the Driver logs. Where do I find the Driver Logs for a SQL Pro Warehouse? The reason I ask is because sometimes in a SQL Editor we get generic error messages like "Task failed while writing row...
I cancel my subscription plan (account created via accounts.databricks.com), without noticing that by doing this, all of my workspaces will be deleted. Is this possible to recover those workspace, and if possible, what should I do to recover them?
Within our team, there are certain (custom) python packages we always use and import in the same way. When starting a new notebook or analysis, we have to import these packages every time. Is it possible to automatically make these imports available ...
@Kaz You can install these libraries using the Libraries section in the Compute. All of the libraries mentioned here would be installed whenever the cluster is spun up.
Is it possible to show the full logs of a databricks job? Currently, the logs are skipped with:*** WARNING: max output size exceeded, skipping output. ***However, I don't believe our log files are more than 20 MB. I know you can press the logs button...
Is there a way to create a calculated field in a dashboard from the data that has been put into it?I have an aggregated dataset that goes into a dashboard, but using an average in the calculation will only work if I display the average by the grouped...
Is there a way to create a calculated field in a dashboard from the data that has been put into it?I have an aggregated dataset that goes into a dashboard, but using an average in the calculation will only work if I display the average by the grouped...
We have a setup where we process sensor data in databricks using pyspark structured streaming from kafka streams, and continuisly write these to delta tables. These delta tables are served through a SQL warehouse endpoint to the users. We also store ...
Hi everyone!Is there any function in Databricks's dashboards to pause/halt visualization before a filter is inputted on the dashboard? Currently, my queries/visualizations are getting too long because of the data, and I want to stop it before a filte...
