I need info about any upcoming events that databricks will provide me a free voucher for the Azure platform architect exam , anyone know the time or a hunch about it please ping me the details. i will be stay tuned at that point of time.................
I'm trying to use the standard Python logging framework in the Databricks jobs instead of print. I'm doing this by using spark._jvm.org.apache.log4j.LogManager.getLogger(__name__)however as I'm running this on serverless, I get the following error [J...
I did the same thing however the logs don't show up on the execution of the tasks so I took aimport logging class LoggerBuilder: def __init__(self, log_level: int = logging.INFO) -> None: self.logger = logging.getLogger() self.l...
I created my table using SCD type 2 in SQL. I need to do full refresh to load all of the data. Whenever I update data in my source table, in my new table scd_target I see only the latest record, history is not being saved. CREATE OR REFRESH STREAMING...
hi @ksenija i got your use case but could you please tell me , what do you mean by "sources_test.sources_test.source"
Where to find log data from PowerBI? I need to find what tables are being used in my PowerBI reports that are pointing to Databricks. I tried system.access.audit but I'm not finding new data when I refresh my report
I'm encountering an issue with incomplete Spark event logs. When I am running the local Spark History Server using the cluster logs, my application appears as "incomplete". Sometime I also see few queries listed as still running, even though the appl...
Hello team,I understand that an automatic upgrade to UC utilizing the UCx tool (Databricks Lab project) is now available to complete this migration from HSM to UC in automate way. But does this tool allow workspace to workspace catalog/artifact migra...
Where i can find notebooks for Data Engineering with Databricks
Hi @michalsadlocha ,You can find them at Slides & DBC section:
On Thursday the 15th August I noticed that several API keys over all of the accounts I run on Databricks had been removed. They had no expire date and some accounts where on a Azure hosted workspace and some on AWS. I would like to hear if you experi...
@Retired_mod Yes this is one of the accounts that have had the token removed. I sent an e-mail about the other account.@ChendaZhangDB Thank you for the idea, this could be the case for 2 tokens that were removed. However I also had 2 tokens removed t...
How can we create external tables in Databricks using DBT Models?
Hi @hrushi512 ,You can try to use location_root config parameter, as they did below:https://discourse.getdbt.com/t/add-location-to-create-database-schema-statement-in-databricks-to-enable-creation-of-managed-tables-on-external-storage-accounts/6894
Hi,I tried to set the spark checkpoint location in a notebook to a folder in a Unity Catalog Volume, with the following command: sc.setCheckpointDir("/Volumes/catalog_name/schema_name/volume_name/folder_name")Unfortunately I receive the following err...
I am facing the same issue on DBR 14.3 and the beta of 15.4.My cluster is using the "Unrestricted" policy and "Single user" access mode set a user which has permission to read and write to the volume. I tested the permissions by writing a small dataf...
Hi Team,We are trying to create DLT pipeline. The scenario is this:- We already have catalog in unity catalog and under that we have schema called test in which we have 17 tables. Now we are trying to create DLT pipeline and copy the data from those ...
Hi Jessy,Thanks for reply. Please find the error message below
Trying to provision a simple workspace. All the code I can find look something like the following, where credential/storage/network resources are created, then referenced. In my case, it's a separate repo and try to reuse existing configurations. I...
Hi @ThisNoName, Based on your description it looks like you're trying to get Databricks account level information for networks and storage configurations. You can easily achieve that through the Databricks account API. Here are the docs: - Get all ...
I am running dbt on a databricks job. It saves all documentation: manifest.json, run_results.json, etc in "Download Artifacts" in a job. I am not able to find out a way to read those in codes, transform and save on databricks. Tried job API. The arti...
Hi Community,My Workflows has been running smoothly since it was created but for the past week, I have been getting this errorI have tried different method and Documentations but nothing seems to work. All the different jobs in my workflow gets this ...
Hello @Jay_Kay Could you please attempt to create a table from this view and re-run your job to verify if it works? Additionally, please ensure that you persist the view dependencies as tables as well. Best Regards, Jéssica Santos
I have a .zip file present on an ADLS path which contains multiple files of different formats. I want to get metadata of the files like file name, modification time present in it without unzipping it. I have a code which works for smaller zip but run...
Here is the code which i am using def register_udf(): def extract_file_metadata_from_zip(binary_content): metadata_list = [] with io.BytesIO(binary_content) as bio: with zipfile.ZipFile(bio, "r") as zip_ref: ...
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group