Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I am not sure if I am missing something, but I just created External Table using External Location and I can still access both data through the table and directly access files from the storage: documentation:https://docs.databricks.com/data-governanc...
I got the answer from the Databricks Support on this.The point which has been mentioned in doc "Once a table is created in a path, users can no longer directly access the files in that path even if they have been given privileges on an external locat...
Looking at the instance profiles we appear to be using AWS IAM roles, however there is a databricks user in our IAM that cloudtrail currently shows its aws access key is being used to describeinstances and describevolumes and I am trying to track dow...
Hi, @ggleason , you can list secrets , ref: https://docs.databricks.com/security/secrets/secrets.html#list-secrets. You can also check on secrets API: https://docs.databricks.com/dev-tools/api/latest/secrets.html
Unable to set x axis range for charts in sql dashboard. The automatic range and padding that is set doesn't match across all plot types. For line charts with date type data on x axis, there is a padding on both sides of the plot. But for bar charts, ...
Hi @John William Databricks clusters use Spark's Standalone cluster manager. Each Databricks cluster has its own standalone Master and Worker processes run inside of the LXC containers and share a lifecycle with the cluster. Each cluster has a singl...
Hello everyone, every day I extract data into xls files but the column position changes every day. Is there any way to find these columns within the file?Here's a snippet of my code.df = spark.read.format("com.crealytics.spark.excel")\
.option("hea...
Hi @welder martins Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
I am trying to utilize the Event Log DLT is keeping updated, I noticed some of the fields are consistently empty/null.In the Event Log, located ".../storage/system/events", I see the field "origin" and there are nested fields within which are empty/n...
Hi @Kristian Foster,The following docs will provide more details on the event log schema. Please refer to this link https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#monitor-pipelines-with-the-delta-live-tables...
Hi,
I'd like to ask you, how much resources do you plan to dedicate to the maintenance/development of the official Databricks Docker images, please? Do you have a view on the longer-term plan for these docker images?
It seems to be maintained, but i...
We were trying to update permissions of Storage credential using DBR 11.2 Shared mode but running into following issue Operation not allowed: GRANT(line 1, pos 0) PFA complete error in file.Is the above issue with cluster permission or changing permi...
Hi FriendsI am following Databricks Customer Academy training material, and created a Databricks service in Azure Trial account and was able to launched a single node cluster there. Could you please guide me on how to do all the hands-on?
Hi All,I have few questions using the community edition 1) max file size that is allowed to be uploaded (data file) in community edition ?2) is XML file supported as well ? Regards,Rakesh
When looking in the standard output of a notebook run in a cluster, we get this message: "To connect another client to this kernel, use: /databricks/kernel-connections-dj8dj93d3d3.json"Is it possible to connect to the databricks ipython kernel and ma...
Hello Everyone,I'm trying to bulk load tables from a SQL server database into ADLS as parquet files and then loading these files into Delta tables (raw/bronze). I had done a one off history/base load but my subsequent incremental loads (which had a d...
I Have a Dataframe stored in the format of delta into Adls, now when im trying to append new updated rows to that delta lake it should, Is there any way where i can delete the old existing record in delta and add the new updated Record.There is a uni...
You should use a MERGE command on this table to match records on the unique column. Delta Lake does not enforce primary keys so if you append only the duplicate ids will appear. Merge will provide you the functionality you desire. https://docs.databr...
