- 3513 Views
- 1 replies
- 0 kudos
Asset bundle build and deploy python wheel with versions
Hi all,I was able to deploy a wheel to the /Shared/ folder from a repository in Gitlab with asset bundles. The databricks.yml looks something like this.artifacts: default: type: whl build: poetry build path: . targets: workspace: h...
- 3513 Views
- 1 replies
- 0 kudos
- 0 kudos
Finally I decided to use AWS Code Artifact and mirror the PyPI, which I think it's a bit cleaner. But your solution looks good too. Thanks!
- 0 kudos
- 2028 Views
- 1 replies
- 2 kudos
Create new workbooks with code
Is is possible to create new notebooks from a notevbook in databricks? I have tried this code. But all of them are generic files, not notebooks.notebook_str = """# Databricks notebook source import pyspark.sql.functions as F import numpy as np # CO...
- 2028 Views
- 1 replies
- 2 kudos
- 2 kudos
Unfortunaly %run does not help me since I can't %run a .py file. I still need my code in notebooks.I am transpiling propriatary code to python using jinja templates. I would like to have the output as notebooks since those are most convenient to edit...
- 2 kudos
- 2896 Views
- 1 replies
- 0 kudos
Resolved! DLT Pipeline Graph is not detecting dependencies
Hi,This is my first databricks project. I am loading data from a UC external volume in ADLS into tables and then split one of the tables into two tables based on a column. When I create a pipeline, the tables don't have any dependencies and this is...
- 2896 Views
- 1 replies
- 0 kudos
- 0 kudos
While re-implementing my pipeline to publish to dev/test/prod instead of bronze/silver/gold, I think I found the answer. The downstream tables need to use the LIVE schema.
- 0 kudos
- 2521 Views
- 1 replies
- 1 kudos
Unpivoting data in live tables
I am loading data from CSV into live tables. I have a live delta table with data like this:WaterMeterID, ReadingDateTime1, ReadingValue1, ReadingDateTime2, ReadingValue2It needs to be unpivoted into this:WaterMeterID, ReadingDateTime1, ReadingValue1...
- 2521 Views
- 1 replies
- 1 kudos
- 1114 Views
- 0 replies
- 0 kudos
How to suppress text output and just show the results in each cell?
How can I turn off displaying all this config text after running cells? I don't need to see all this text in every cell unless there is an error and it causes readability issues and makes the notebook scrolling tedious.
- 1114 Views
- 0 replies
- 0 kudos
- 2810 Views
- 2 replies
- 0 kudos
time travel with DLT
Needed some help with Time Travel with Delta Live tables We were trying to figure out if we can go in and alter the history on this table, and what would happen to data that we mass upload? By this we mean we have data from the past that we would ...
- 2810 Views
- 2 replies
- 0 kudos
- 0 kudos
Delta Live Tables leverage Delta Lake, or Delta Tables. Delta tables, through transactions (e.g. insert, update, delete, merges, optimization) create versions of said Delta Table. Once a version is created it cannot be altered, it is immutable. Yo...
- 0 kudos
- 2644 Views
- 1 replies
- 0 kudos
Upgrade Spark version 3.2 to 3.4+
Hi Team,We would like to upgrade from Spark version 3.2 to 3.4+ (Databricks Runtime - 10.4 to 12.2/13.3)We would like to understand how complex upgradation is this and challenges which we face? what are the technical steps and precautions we need to ...
- 2644 Views
- 1 replies
- 0 kudos
- 2029 Views
- 0 replies
- 0 kudos
Customer Managed Keys in Databricks (AWS)
Hi Databricks Team,Could you please provide me the detailed steps on how to be enabled customer managed keys in databricks (AWS) Account, if there is any video on it that would be great helpful.Regards,Phanindra
- 2029 Views
- 0 replies
- 0 kudos
- 4198 Views
- 0 replies
- 0 kudos
Spark Streaming Issues while performing left join
Hi team,I'm struck in a Spark Structured streaming use-case.Requirement: To read two streaming data frames, perform a left join on it and display the results. Issue: While performing a left join, the resultant data frame contains only rows where ther...
- 4198 Views
- 0 replies
- 0 kudos
- 1029 Views
- 0 replies
- 0 kudos
AI uses
Delve into the transformative realm of AI applications, where innovation merges seamlessly with technology's limitless possibilities.Explore the multifaceted landscape of AI uses and its dynamic impact on diverse industries at StackOfTuts.
- 1029 Views
- 0 replies
- 0 kudos
- 1762 Views
- 1 replies
- 0 kudos
Load GCP data to Databricks using R
I'm working with Databricks and Google Cloud in the same project. I want to load specific datasets stored in GCP into a R notebook in Databricks. I currently can see the datasets in BigQuery. The problem is that using the sparklyr package, I'm not ab...
- 1762 Views
- 1 replies
- 0 kudos
- 3006 Views
- 0 replies
- 1 kudos
Best AI Art Generator
AI art generator uses artificial intelligence to create captivating artworks, redefining the boundaries of traditional creativity and enabling endless artistic possibilities.AI photo restoration is a groundbreaking technology that employs artificial ...
- 3006 Views
- 0 replies
- 1 kudos
- 2551 Views
- 1 replies
- 0 kudos
Multi Customer setup
We are trying to do POC to have shared resource like compute across multiple customer, Storage will be different, Is this possible ?
- 2551 Views
- 1 replies
- 0 kudos
- 6386 Views
- 0 replies
- 0 kudos
Alter table
Hi Team, Could you please suggest, Do we have an alternate approach to alter the table instead of creating a new table and copying the data as part of the deployment.
- 6386 Views
- 0 replies
- 0 kudos
- 2286 Views
- 1 replies
- 0 kudos
Constraint options usage
I am trying to use the constraints options:NOT ENFORCEDDEFERRABLEINITIALLY DEFERREDNORELYHowever it seems I am not able to use them successfully. When I try to use them with PRIMARY KEYS (not sure if it is possible), I am not able to enforce any key....
- 2286 Views
- 1 replies
- 0 kudos
- 0 kudos
BTW the forum is bugged I can't paste code..
- 0 kudos
