- 370 Views
- 1 replies
- 0 kudos
Hi everyone,I’m looking to migrate ML model experiments from a source Databricks workspace to a target workspace. Specifically, I want to use Python and the available REST APIs for this process.Can anyone help me on this!Thanks in advance!
- 370 Views
- 1 replies
- 0 kudos
Latest Reply
You can use https://github.com/mlflow/mlflow-export-import utility.
The example given below doesn't use Python but uses CLI and CICD pipeline to do the same.
https://medium.com/@gchandra/databricks-copy-ml-models-across-unity-catalog-metastores-188...
- 1709 Views
- 2 replies
- 3 kudos
Hi, I'm super new to Databricks. I'm trying to do a little API scripting against my company's DB instance.I have this supersimple python (ver 3) which is meant to run a remote host. The script tries to a simple SQL query against my Databricks instan...
- 1709 Views
- 2 replies
- 3 kudos
Latest Reply
@gchandra Yes! This is the documentation I was seeking! Thank you so much
1 More Replies
- 1804 Views
- 3 replies
- 2 kudos
Hey guys, I have been using Postman for quite some time now and getting disappointed recently and want to make a swtich. Is there something better than Postman? I've heard about that APIDog is much easier to use with a much better UI, and support all...
- 1804 Views
- 3 replies
- 2 kudos
by
Phani1
• Valued Contributor II
- 841 Views
- 1 replies
- 0 kudos
Hi All,We are facing a situation where our data source is Snowflake, and the data is saved in a storage location(adls) in parquet format. However, the tables or data lack a date column or any incremental column for performing incremental loads to Dat...
- 841 Views
- 1 replies
- 0 kudos
Latest Reply
Ideally you would have some change tracking system (cdc f.e.) on the source tables (Streams in the case of Snowflake, Introduction to Streams | Snowflake Documentation).But that is not the case.So I think you approach is ok. You cannot track what is...
- 2227 Views
- 2 replies
- 1 kudos
I'm setting up a job in the Databricks Workflow UI and I want to pass a dynamic parameter, like the current date (run_date), each time the job runs.In Azure Data Factory, I can use expressions like @utcnow() to calculate this at runtime. However, I w...
- 2227 Views
- 2 replies
- 1 kudos
Latest Reply
As szymon mentioned, dynamic parameter values exist, but the functionality is still far from what Data Factory has to offer.I am pretty sure though that this will be extended.So for the moment I suggest you do the value derivation in data factory, an...
1 More Replies
- 5339 Views
- 1 replies
- 0 kudos
Hello,Would anyone know if it is possible to create a delta table using Python that includes a column that is generated by default as identity (identity column for which the value inserted can be manually overriden)?There seems to be a way to create ...
- 5339 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @oleprince ,As far as I know, it's not possible yet to create tables with Identity columns using pyspark (DeltaTable api). You can create generated columns, but Identity columns are not allowed.The only way to achieve this is through Spark Sql.
- 1830 Views
- 7 replies
- 1 kudos
Hey, I am new to Databricks, and I am trying to test the mlops-stack bundle. Within that bundle there is a feature-engineering workflow and I have a problem to make it run. The main problem is the following.the bundle specified the target to be $bund...
- 1830 Views
- 7 replies
- 1 kudos
- 585 Views
- 1 replies
- 0 kudos
Scaling XGBoost and LightGBM models to handle exceptionally large datasets—those comprising billions to tens of billions of rows—presents a formidable computational challenge, particularly when constrained by the limitations of in-memory processing o...
- 585 Views
- 1 replies
- 0 kudos
Latest Reply
Well, that's a superb article! Thank you for this great information, you write very well which I like very much. I am really impressed by your post. run 3
by
Phani1
• Valued Contributor II
- 754 Views
- 1 replies
- 0 kudos
Hi All,We have a situation where we are collecting data from different Oracle instances.The customer is using Oracle GoldenGate to replicate this data into a storage location.From there, we can use Auto Loader or Delta Live Tables to read Avro files ...
- 754 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Phani1 ,In my opinion this is really good setup. You have push scenario where Oracle GoldenGate is responsible for delivering data into storage, so you don't have to bother about extraction part. And autoloader is the best choice when it comes t...
by
Phani1
• Valued Contributor II
- 253 Views
- 0 replies
- 0 kudos
Hi All,How can we connect Databricks Delta Lake to Essbase in OCI? We know that Essbase supports JDBC/ODBC. Is it possible to use Python or PySpark to read from Delta Lake and load the data into Essbase? I think using JDBC/ODBC might affect performan...
- 253 Views
- 0 replies
- 0 kudos
by
Phani1
• Valued Contributor II
- 549 Views
- 0 replies
- 0 kudos
Hi All,We are establishing a connection from Denodo to Databricks. During the development phase, we utilized a personal access token associated with developer account. However, this approach is not considered a best practice for production environm...
- 549 Views
- 0 replies
- 0 kudos
- 1162 Views
- 2 replies
- 1 kudos
I'm trying to create a delta live table from a Google PubSub stream.Unfortunately I'm getting the following error:org.apache.spark.sql.streaming.StreamingQueryException: [PS_FETCH_RETRY_EXCEPTION] Task in pubsub fetch stage cannot be retried. Partiti...
- 1162 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @Retired_mod, it was indeed a permissions issue. Nevertheless, I must admit that the error message is slightly misleading.Thanks.
1 More Replies
by
Surajv
• New Contributor III
- 1437 Views
- 2 replies
- 0 kudos
Hi community,Assume I generate a personal access token for an entity. Post generation, can I restrict the access of the entity to specific REST APIs? In other words, consider this example where once I use generate the token and setup a bearer token b...
- 1437 Views
- 2 replies
- 0 kudos
Latest Reply
@Surajv You have to rely on access control settings on resources and entities (users or service principals or create some cluster policies), rather than directly restricting the API endpoints at the token level.Note: API access based on fine-grained ...
1 More Replies
- 1350 Views
- 1 replies
- 1 kudos
When you're using a very large notebook sometimes it gets frustrating scrolling through all the code blocks. It would be nice to have a few additional options to make this easier. 1) Add a collapse all code cells button to the top.2) Add a collapse a...
- 1350 Views
- 1 replies
- 1 kudos
- 726 Views
- 1 replies
- 0 kudos
Does a queued databricks job incur cost?
- 726 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @qwerty3 ,No, it does not. When a job is queued (waiting for an available cluster or resources), there is no compute usage, so no charges are incurred for Databricks units (DBUs) or cloud infrastructure (VMs). The queued state is essentially a wai...
