- 217 Views
- 1 replies
- 0 kudos
Migrating ML Model Experiments Using Python REST APIs
Hi everyone,I’m looking to migrate ML model experiments from a source Databricks workspace to a target workspace. Specifically, I want to use Python and the available REST APIs for this process.Can anyone help me on this!Thanks in advance!
- 217 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use https://github.com/mlflow/mlflow-export-import utility. The example given below doesn't use Python but uses CLI and CICD pipeline to do the same. https://medium.com/@gchandra/databricks-copy-ml-models-across-unity-catalog-metastores-188...
- 0 kudos
- 991 Views
- 2 replies
- 1 kudos
How to Pass Dynamic Parameters (e.g., Current Date) in Databricks Workflow UI?
I'm setting up a job in the Databricks Workflow UI and I want to pass a dynamic parameter, like the current date (run_date), each time the job runs.In Azure Data Factory, I can use expressions like @utcnow() to calculate this at runtime. However, I w...
- 991 Views
- 2 replies
- 1 kudos
- 1 kudos
As szymon mentioned, dynamic parameter values exist, but the functionality is still far from what Data Factory has to offer.I am pretty sure though that this will be extended.So for the moment I suggest you do the value derivation in data factory, an...
- 1 kudos
- 4668 Views
- 1 replies
- 0 kudos
Delta table definition - Identity column
Hello,Would anyone know if it is possible to create a delta table using Python that includes a column that is generated by default as identity (identity column for which the value inserted can be manually overriden)?There seems to be a way to create ...
- 4668 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @oleprince ,As far as I know, it's not possible yet to create tables with Identity columns using pyspark (DeltaTable api). You can create generated columns, but Identity columns are not allowed.The only way to achieve this is through Spark Sql.
- 0 kudos
- 343 Views
- 1 replies
- 0 kudos
Addressing Memory Constraints in Scaling XGBoost and LGBM: A Comprehensive Approach for High-Volume
Scaling XGBoost and LightGBM models to handle exceptionally large datasets—those comprising billions to tens of billions of rows—presents a formidable computational challenge, particularly when constrained by the limitations of in-memory processing o...
- 343 Views
- 1 replies
- 0 kudos
- 0 kudos
Well, that's a superb article! Thank you for this great information, you write very well which I like very much. I am really impressed by your post. run 3
- 0 kudos
- 1064 Views
- 1 replies
- 1 kudos
Feature Request: GUI: Additional Collapse options
When you're using a very large notebook sometimes it gets frustrating scrolling through all the code blocks. It would be nice to have a few additional options to make this easier. 1) Add a collapse all code cells button to the top.2) Add a collapse a...
- 1064 Views
- 1 replies
- 1 kudos
- 1 kudos
this feature please
- 1 kudos
- 557 Views
- 1 replies
- 0 kudos
Resolved! Does a queued databricks job incur cost?
Does a queued databricks job incur cost?
- 557 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @qwerty3 ,No, it does not. When a job is queued (waiting for an available cluster or resources), there is no compute usage, so no charges are incurred for Databricks units (DBUs) or cloud infrastructure (VMs). The queued state is essentially a wai...
- 0 kudos
- 1504 Views
- 11 replies
- 1 kudos
Resolved! databricks Asset Bundle
i have come accross a documentation on asset bundles long back whcih states that when you typedatabricks bundle initit gives us option to choose a project type. But i see the below error when i do that i see the below erroris there a way, i can take ...
- 1504 Views
- 11 replies
- 1 kudos
- 653 Views
- 1 replies
- 2 kudos
column mask on <tinyint>Y columns gives error
My table breaks when I try to mask a column with a name like `<tinyint>Y` -- Create a table with a masked column> CREATE FUNCTION mask_int_col(col_val INTEGER) RETURN CASE WHEN is_member('HumanResourceDept') THEN col_val ELSE CAST(NULL as INTEGER) EN...
- 653 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @DW ,I have replicated your scenario and encountered the same error when applying a column mask to a column named 1Y in Databricks SQL.In short, it makes sense simply to follow Databricks documentation and use the SQL naming conventions, so that c...
- 2 kudos
- 356 Views
- 1 replies
- 0 kudos
Databricks Pyspark filter several columns with similar criteria
I am querying a table from the Databricks Catalog which I have to filter several columns with the same criteria. below is what I have created so far. I have 10 columns that I have filter with a set of criteria from (dx_list1) and another 10 that I ...
- 356 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @abueno ,As I understand the logic you want to implement is:1. For every pair of columns:First Column (DX_i): Must be in dx_list1Second Column (DX_{i+1}): Must be in dx_list22. The condition for each pair is:col('DX_i').isin(dx_list1) OR col('DX_{...
- 0 kudos
- 455 Views
- 1 replies
- 0 kudos
Set up compute policy to allow installing python libraries from a private package index
In our organization, we maintain a bunch of libraries we share code with. They're hosted on a private python package index, which requires a token to allow downloads. My idea was to store the token as a secret which would then be loaded into a cluste...
- 455 Views
- 1 replies
- 0 kudos
- 0 kudos
I figured it out, seems like secrets can only be loaded into environment variables if the content is the secret and nothing else:"value": "{{secrets/global/arneCorpPyPI_token}}" # this will work"value": "foo {{secrets/global/arneCorpPyPI_toke...
- 0 kudos
- 1328 Views
- 3 replies
- 1 kudos
How to retrieve Spark Session inside java jar library installed on Cluster
I have a java app in form of jar package. This jar is installed on a Databricks cluster. This jar package reads and writes to few tables in databricks. In order to achieve that, I need SparkSession available in the code. Given that spark session is a...
- 1328 Views
- 3 replies
- 1 kudos
- 201 Views
- 0 replies
- 0 kudos
Course registration issue
Hi All,I'm trying to register for "Spark Developer course" which is free course but I don't see this course under the course catalogue and if I try to directly access using a link I get 403 forbidden error. Can anyone help me figure out the issue.
- 201 Views
- 0 replies
- 0 kudos
- 1137 Views
- 7 replies
- 0 kudos
Unable to install the Databricks in AWS
I have followed the instructions given in the following link https://docs.databricks.com/en/admin/workspace/quick-start.html#use-aws-quickstart-to-create-a-databricks-workspace%22There was an error in CFT while deploying the workspace. Please help[ER...
- 1137 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @rseshadri ,I'm more familiar with Azure, but this error is most commonly caused by entering the wrong credentials in the Cloudformation template.Try to launch the Quickstart again from the Databricks Account Console and double check that you ente...
- 0 kudos
- 320 Views
- 2 replies
- 1 kudos
How to delete object permissions using REST API
We have REST API methods to set,update,Get object permissions documented here https://docs.databricks.com/api/workspace/permissionsIs there a way to revoke the permissions which are granted through REST API
- 320 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you for your response!We have a situation where we assigned CAN_BIND permissions to a group for a service principal using the following:$ vim grant-service-principal.json{"access_control_list": [{"user_name": "{username}","permission_level": "C...
- 1 kudos
- 1594 Views
- 3 replies
- 0 kudos
How to Read Data from Databricks Worker Nodes in Unity Catalog Volume
I am currently working on a similarity search use case where we need to extract text from PDF files and create a vector index. We have stored our PDF files in a Unity Catalog Volume, and I can successfully read these files from the driver node.Here's...
- 1594 Views
- 3 replies
- 0 kudos
