Is it possible to show the full logs of a databricks job? Currently, the logs are skipped with:*** WARNING: max output size exceeded, skipping output. ***However, I don't believe our log files are more than 20 MB. I know you can press the logs button...
Hi @Kaz , it's not possible to display the complete logs of a Databricks job in the job overview if the log output size has been exceeded. Databricks have a limit on the size of the output logs that can be displayed in the job overview. If the output...
Is there a way to create a calculated field in a dashboard from the data that has been put into it?I have an aggregated dataset that goes into a dashboard, but using an average in the calculation will only work if I display the average by the grouped...
Hi @SimonMcCor , Yes, you can create a calculated field in a dashboard from the data that has been put into it. In Databricks, you can perform this operation within the notebook that feeds data into your dashboard.
If you want to request a new featur...
We have a setup where we process sensor data in databricks using pyspark structured streaming from kafka streams, and continuisly write these to delta tables. These delta tables are served through a SQL warehouse endpoint to the users. We also store ...
Hi @Erik ,
- Use Delta Lake to optimize tables by coalescing small files into larger ones- Use the OPTIMIZE command on Delta Lake tables to improve write speed and reduce the number of small files- Utilize ZORDER for multi-dimensional clustering to ...
Databricks asset bundles, now in Public Preview, enable end-to-end data, analytics, and ML projects to be expressed as a collection of source files. This makes it simpler to apply data engineering best practices such as source control, code review, t...
Getting this error in dbt when trying to run a query. Not happening in the actual SQL warehouse in Databricks. Is this a bug? Can only find source code when I search 'DatabricksSQLCursorWrapper' but no documentation or information otherwise.
Hi @colinsorensen, Based on the given information, it appears that the error you are encountering is related to accessing cloud storage in Databricks. The error message suggests that the cluster does not have the necessary permissions to access the s...
Does anybody know whether we could customise the dates showing in the calendar selection in sql editor/dashboard?My query has a time frame in a particular period, however when I use DateRange parameter in sql editor, it could allow users to choose th...
Hi @hehuan-yu-zen , Yes, you can customize the dates showing in the calendar selection in the SQL editor/dashboard. The Date Range, Date and Time Range, and Date and Time Range (with seconds) parameters now support the option to designate the startin...
Hi everyone,We are at the moment stumbeling upon a big challenge with loading data into PowerBI. I need some advice!To give a bit of conext: we introduced Databricks instead of Azure Synapse for a client of ours. We are currently busy with moving all...
Currently our solution to this problem is using a Personal Access Token as authentication method. I stumbled upon the problem that when the dataset is scheduled via PowerBI it went back to OAuth authentication. Still checking if the problem is stayin...
Hi allWe are having some alarming issues regarding a script that yields different output when running on SQL vs Notebook. The correct output should be 8625 rows which it is in the notebook, but the output in Databricks SQL is 156 rows. The script use...
UPDATE:I think we have identefied and solved the issue. It seems like using LAST with Databricks SQL requires to excplicitly be careful about setting the "ignoreNull" argument and also be careful about the correct datatype. I guess this is because of...
Did you know the default timeout setting for SQL #databricks Warehouse is two days?The default timeout can be too long for most use cases. You can easily change this for your session or in the general SQL warehouse configuration.
Hi @Hubert-Dudek, I want to extend my heartfelt appreciation to our esteemed Community Member for consistently delivering outstanding company updates that are both informative and insightful. Your dedication to keeping the community well-informed and...
We have previously been able to access our Databricks cluster in R using ODBC but it stopped working a couple of months ago and now i can't get it to connect.I've downloaded the latest drivers and added the right information in odbc/odbcinst files bu...
Hi @uberweiss, Based on the error message you are receiving, the issue might be related to the server host or port specified for the connection. Here are a few suggestions to troubleshoot this issue.
1. Verify Server Details: The error message sugges...
I have created a filter through the editing part in the dashboard, but when I drag this widget it cannot locate in other places except the top area. I know the filter in the dashboard has the unction to control the whole queries/ whole visualizations...
We're creating a report with Power BI using data from our AWS Databricks workspace. Currently, I can view the report on Power BI (service) after publishing. Is there a way to change the data source connection, e.g. if I want to change the data source...
Hi @Kaniz,Thanks for your reply! It seems possible, but I'm stuck on step 3, since `extensionDataSourcePath` is greyed out. Everything else is editable. What permissions do I need in order to update that field?
Hi Guys,I have a JSON as the below structure where the key is as decimal.{ "5.0": { "a": "15.92", "b": 0.0, "c": "15.92", "d": "637.14" }, "0.0": { "a": "15.92", "b": 0.0, "c": "15.92", "d": "637.14" } }schema_of_json returns the following:STRUCT<`0....
