For any new Data Engineering aspirant, it has always been a difficult where to start the learning journey. I faced this challenge a decade ago. In order to help new aspirants I created a series of medium article for new learners. I hope it brings mor...
When using Asset Bundles to deploy jobs, how does one get the job ID of the resources that are created?I would like to deploy some jobs through asset bundles, get the job IDs, and then trigger these jobs programmatically outside the CI/CD pipeline us...
Refer this answer and this can be a solution to above scenario -> https://community.databricks.com/t5/data-engineering/getting-job-id-dynamically-to-create-another-job-to-refer-as-job/m-p/102860/highlight/true#M41252
Any help to understand why it's showing 'null' instead of the date value? It's showing null only for 12:00:00 AM and for any other values it's showing date correctlyTO_DATE("12/30/2022 12:00:00 AM", "MM/dd/yyyy HH:mm:ss a") AS tsDate
Hi @David_Billa, Can you try with: TO_TIMESTAMP("12/30/2022 12:00:00 AM", "MM/dd/yyyy hh:mm:ss a") AS tsDate The issue you are encountering with the TO_DATE function returning null for the value "12:00:00 AM" is likely due to the format string not ma...
I have a field stored as a string in the format "12/30/2022 10:30:00 AM"If I use the function TO_DATE, I only get the date part... I want the full date and time.If I use the function TO_TIMESTAMP, I get the date and time, but it's assumed to be UTC, ...
use from_utc_timestamp(to_timestam("<string>", <format>),<timezone>)
HiI am encountering the following error when writing a DLT pipeline. Here is my workflow:Read a bronze delta tableCheck Data Quality RulesWrite clean records to a silver table with defined schema. I use TRY_CAST for columns where there is mismatch be...
Hi @Svish ,You have one line that differs:JOB_CERTREP_CONTRACT_INT: string (nullable = true) vs. JOB_CERTREP_CONTRACT_NUMBER: string (nullable = true)
I am trying to deploy a dbt task as part of a databricks job using databricks asset bundles.However, there seems to be a clash that occurs when specifying a job that includes a dbt task that causes a bizarre failure.I am using v0.237.0 of the CLI.Min...
Thanks for providing that whole example, it was really easy to fiddle with. I think I've found your solution. Update the original two tasks on the job (if you want to keep them) like this: tasks: - task_key: notebook_task job...
Hello Community,I have a specific scenario regarding Unity Catalog and workspace deletion that I'd like to clarify:Current Setup:Two DataBricks workspaces: W1 and W2Single Unity Catalog instanceCatalog1: Created in W1, shared and accessible in W2Cata...
Hi @HoussemBL When you delete a Databricks workspace, it does not directly impact the Unity Catalog or the data within it. Unity Catalog is a separate entity that manages data access and governance across multiple workspaces. Here’s what happens in ...
We have multiple Databricks instances, one per environment (Dev-UK, Live-UK Live-EU, Live-US, etc), and we would like to create dashboards to present stats on our data in each of these environments. Each of these environments also has a differently n...
Hello, as you have mentioned you could create an script in Python that uses the api call https://docs.databricks.com/api/workspace/lakeview/create to generate the dashboard for each environment, the process to create the visualizations will be comple...
In a notebook attached to.a normal personal cluster I can successfully do this:%fs ls file:/Workspace/Repos/$userName/$repoName/$folderNameWhen I run an init-script on a UC volume that the does the same thing, I'm getting this error:ls: cannot access...
Hi @dollyb, Can you try with just "ls /Workspace/Repos/my_user_name@company.com/my_repo_name/my_folder_name" I'm not sure dbutils will be useful in an init script, I will try to test it out
Hello everyone,In my Databricks partner academy account, there is no course material while it should be under the lesson video. How can I resolve this problem? Does anyone else face the same problem? I had submitted a ticket to ask Databricks team bu...
I am still having this problem, cant find the slides and DLC for Data Engineering with Databricks
Outside of running jobs with different users, is there any way for me to run a notebook (or even better a cell within a notebook) as either a different user or a specific role that is not my user default?I'm trying to find an easy way to test data ma...
Unfortunately there is no direct way to run a notebook using additional principal, only option is to set up a notebook job task and on the Run As specify the principal that will run the job that can be a user or a Service Principal.
Hello Team,Is it possible to share a Dashboard with a user inside the organization that doesn't have a Data-bricks account?Assign a cluster to one notebook/dashboard and share it inside the organization with a SSO login possibility?Suppose we want to...
Hello, thank you for this since I am also looking for this functionality. However, doing more digging with my engineers we found that this feature may allow us to share dashboards to users not in the Workspace, but it still requires the user to have ...
If anyone has example code for building a CDC live streaming pipeline generated by AWS DMS using import dlt, I'd love to see it.I'm currently able to see the parquet file starting with Load on the first full load to S3 and the cdc parquet file after ...
There is a blogpost for this that includes example code that you can find here
How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace Private linkIn your storage accounts please go to “Networking” -> “Private endpoint connections” and click Add Private Endpoint.It is important to add private links in ...
Hi, I would like to se STATEMENT_TIMEOUT for a specific SQL warehouse and not on a global level.How would I do that?P.s. I would like to avoid it on a session level, just one-time configuration for a given SQL warehouse.
Unfortunately we do not support that. We only support Global and Session level settings. We have an internal feature request for this (DB-I-6556 ) but it has not been prioritized in the Roadmap.
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
