- 6604 Views
- 5 replies
- 1 kudos
Connecting from Databricks to Network Path
Hi All,Will appreciate if someone can help me with some references links on connecting from Databricks to external network path. I have Databricks on AWS and previously used to connect to files on external network path using Mount method. Now Databri...
- 6604 Views
- 5 replies
- 1 kudos
- 1516 Views
- 3 replies
- 0 kudos
Databrick select from web address that returns JSON
Hi I'm in a data bricks notebook and want to select from a web site that returns json. For example this web site http://ergast.com/api/f1/2004/1/results.jsonwill return some JSON. (example only) Can i do the following or Do you need to use python. (...
- 1516 Views
- 3 replies
- 0 kudos
- 0 kudos
yes, you could use SQL to read the data from the volume:%sql SELECT * FROM json.`/Volumes/demo/raw/files/ergast/my_results.json`You still have to flatten the result.The thing with the shell was just an example. Wouldn't recommend that, just use pytho...
- 0 kudos
- 8955 Views
- 7 replies
- 8 kudos
Databricks Jobs and CICD
Hi, We currently leverage Azure DevOps to source control our notebooks and use CICD to publish the notebooks to different environments and this works very well. We do not have the same functionality available for Databricks jobs (the ability to sourc...
- 8955 Views
- 7 replies
- 8 kudos
- 8 kudos
To manage Databricks jobs within a DevOps pipeline, start by exporting the job configuration as a JSON file from the Databricks workspace. Parameterize this JSON by replacing environment-specific values with placeholders. Integrate the parameterized ...
- 8 kudos
- 1544 Views
- 1 replies
- 1 kudos
I attended the Advanced Machine Learning with Databricks training last week virtually I am still unable to get the day 2 session videos of any of the...
I attended the Advanced Machine Learning with Databricks training last week virtually I am still unable to get the day 2 session videos of any of the Instructor led Paid Trainings. They are supposed to be available for replay with in 24 hours but I ...
- 1544 Views
- 1 replies
- 1 kudos
- 1 kudos
I have the same problem.
- 1 kudos
- 1402 Views
- 2 replies
- 0 kudos
Data processing metrics
Dear all,What are some proven ways of capturing data processing metrics (number of rows processed/updated/inserted, number of micro-batches etc etc) in a PySpark/SQL code based notebook irrespective of the fact it uses auto-loader, structured streami...
- 1402 Views
- 2 replies
- 0 kudos
- 0 kudos
@Alberto_Umana thanks for the reply. without the current code change, is it possible to extract these metrics from logs?
- 0 kudos
- 3865 Views
- 10 replies
- 1 kudos
My DABS CLI Deploy call not generating a .tfstate file
Hi Community,I'm running into an issue, when executing Databricks CLI Bundle Deploy I dont get the Terraform State file (.tfstate). I know that I should get one but even when defining the state_apth on my YAML (.yml) DABS file I still do not get it.D...
- 3865 Views
- 10 replies
- 1 kudos
- 5881 Views
- 4 replies
- 2 kudos
[AWS] How do you replace the Account Admin?
I need to remove an older admin that previously set up the Databricks Account. However, I get an error (even through I am also an Account Admin).How do I replace a prior account admin? Or at least remove their admin status and/or disable the accoun...
- 5881 Views
- 4 replies
- 2 kudos
- 2 kudos
Was this ever fixed as I have got the exact same issue as our original AWS Account Admin has left the business and I now cannot remove his Account Admin role (but I have disabled access etc)!
- 2 kudos
- 3857 Views
- 4 replies
- 0 kudos
Possibilities and Limitations of Delta Live Tables (DLT) with Direct Publish mode
Hi all,I am implementing a DLT table with the new Direct Publish feature which is still in Private Preview.Is it a limitation of DLT with Direct Publish that you can not query the event_log of the DLT?When I use this query:SELECT * FROM event_log('pi...
- 3857 Views
- 4 replies
- 0 kudos
- 0 kudos
If any one is interested how to access the event_log for DLTs that are writing to different schemas:The approach that was described above was not working anymore - at least for us.But we discovered this post:https://stackoverflow.com/questions/794329...
- 0 kudos
- 12856 Views
- 4 replies
- 1 kudos
Resolved! Connect and process Azure analysis services
How do I connect to Azure analysis services from databricks? I need to process the tabular model from databricks. I tried to use adodbapi. while connecting it is failing with error message "windows com error dispatch adodb.connection". please help
- 12856 Views
- 4 replies
- 1 kudos
- 1 kudos
I got another use case now "to run dax against Azure Analysis Services model" from AWS databricks. I tried above suggestion from "Jun Yang" and it is erroring out after 30 seconds with the exception that "Login timeout is expired"
- 1 kudos
- 3978 Views
- 2 replies
- 1 kudos
Multiple DLT pipelines same target table
Is it possible to have multiple DLT pipelines write data concurrently and in append mode to the same Delta table? Because of different data sources, with different data volumes and required processing, we would like to have different pipelines stream...
- 3978 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi I wanted to understand how you set "pipelines.tableManagedByMultiplePipelinesCheck.enabled" to true? I'm not able to figure out how and where you set this param
- 1 kudos
- 5140 Views
- 4 replies
- 0 kudos
Resolved! Legacy Pipeline?
What is this legacy pipeline warning? I cannot find anything related on the internet and related forums. Thanks in advance.
- 5140 Views
- 4 replies
- 0 kudos
- 0 kudos
Ah I've found the issue. There is one more diff:This is the new key-value for target schema: "schema": "dev", This is the old key-value for target schema: "target": "prod" After changing the key from "target" to "schema", the warning has gone....
- 0 kudos
- 4459 Views
- 2 replies
- 1 kudos
Resolved! Configure Soft-Delete Retention Period of Unity Catalog Managed Table
According to the docs,When a managed table is dropped from Unity Catalog, its underlying data is deleted from your cloud tenant within 30 days.Can I configure the retention period? Is executing DELETE FROM my_table followed by VACUUM my_table RETAIN ...
- 4459 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @guiferviz ,Yes, that is correct! The only way is to use DELETE and VACCUM commands. For managed tables, the default retention period is set to 30‑days. Unity Catalog enforces this to provide an additional safety net against accidental data loss, ...
- 1 kudos
- 69835 Views
- 5 replies
- 3 kudos
Resolved! How to set a variable and use it in a SQL query
I want to define a variable and use it in a query, like below: %sql SET database_name = "marketing"; SHOW TABLES in '${database_name}';However, I get the following error:ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near ''''(line 1, pos...
- 69835 Views
- 5 replies
- 3 kudos
- 3 kudos
CJS had the best answer by virtue of it being code-based rather than widget-based. In a notebook where the value of the variable must continually be reset, widgets are suboptimal. That said, the easiest way is to wrap the code in python:%pyvar1 = s...
- 3 kudos
- 18354 Views
- 4 replies
- 0 kudos
Using a Virtual environment
Hi All,We are working on training NHits/TFT (a Pytorch-forecasting implementation) for timeseries forecasting. However, we are having some issues with package dependency conflicts.Is there a way to consistently use a virtual environment across cells ...
- 18354 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @sshssh Databricks support Notebook scoped environments for databricks runtime (regular and ML) Kindly please check blog post here for ways to install Notebook scoped libraries in databricks Regards
- 0 kudos
- 2412 Views
- 1 replies
- 0 kudos
Update target column with optional source columns
I am trying to load on prem transaction log table and update a databricks table.In Source table,the column reqdetails table hold all the information and expect ProductID remaining columns are dynamic(not all columns exists in the request) all time th...
- 2412 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi RucHow are you doing today?, as per my understanding, Since your source table has a ReqDetails column with dynamic key-value pairs and only ProductID is fixed, you can parse the ReqDetails column dynamically and merge it into the target table usin...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
